Ambient infrared detection in solid state sensors

ABSTRACT

A solid state imaging device includes an array of active pixels and an infrared cut filter formed over the sensor. Optionally, a slot in the infrared cut filter allows infrared illumination to reach the sensor to be detected by pixels covered by a visually opaque filter and surrounded by pixels of special types that limit charge leakage and enable high dynamic range sensing of infrared illumination. A ratio of average infrared signal to average brightness indicates an amount of infrared illumination reaching the imaging device.

TECHNICAL FIELD OF THE INVENTION

The present invention relates generally to optical devices and inparticular the present invention relates to image sensors.

BACKGROUND OF THE INVENTION

Image sensors are used in many different types of electronic devices tocapture an image. For example, consumer devices such as video camerasand digital cameras as well as numerous scientific applications useimage sensors to capture an image. An image sensor is comprised ofphotosensitive elements that sense incident illumination and produces anelectrical signal indicative of that illumination. Each photosensitiveelement is typically referred to as a picture element or pixel.

Image sensors include charge coupled devices (CCD) and complementarymetal oxide semiconductor (CMOS) sensors. Imaging sensors may be capableof capturing grayscale and/or color images. Imaging sensors capable ofcapturing color images often employ a color filter array (CFA) toseparate visible illumination of different color. For example, each ofthe pixels can be covered with a red, green, or blue filter according toa specific pattern. For example, the Bayer pattern has a repeatingpattern of an alternating row of green and red and an alternating row ofblue and green. As a result of the filtering, each pixel of the colorimage captured by a CMOS sensor with CFA responds to only theillumination of wavelengths determined by the color filter of the threeprimary light colors.

CFA color filters are typically transparent to infrared illumination. Invideo and still digital cameras, an infrared-cutoff filter is typicallyplaced in front of the sensor to ensure that only visible wavelengthillumination reaches the sensor. This is done to make possible accuratecolor reproduction in captured images.

White balance is the compensation of an image for variation in colortemperature of scene illuminant. Images are adjusted in such a way thatgray objects will look gray when displayed on a standard output devicewith a predefined color temperature. Other colors displayed on theoutput device will also appear as they appeared during image capture.Without white balance, images captured using illuminants having a lowtemperature will exhibit a reddish cast when displayed on a standard RGBmonitor with the color temperature of the white point of 6500K.Illuminants of higher color temperature can create a bluish cast. Thehuman eye can compensate for the different color temperatures. Both filmand digital cameras, however, need to correct for different lightsources in order to render an image in its true colors.

In a film camera, the color correction can be accomplished manually bythe camera user attaching a color correction filter over the lens. Adigital camera can correct the color temperature automatically byestimating the color temperature of the scene illuminant. Then theentire image is adjusted by the difference between the scene illuminantcolor temperature and the color temperature of the white point of thetarget output device (e.g., a standard display). One problem with thisapproach is that the white balance algorithm has to be informed as tothe correct color temperature of the scene illuminant.

Various techniques exist in the art to determine the color temperatureand spectral content of the illuminant. The color temperature of theilluminant can be reliably identified if one knows which objects in thescene are white. Determining the current lighting conditions in adigital camera may be difficult since cameras cannot reliablyautomatically identify which objects in the scene are white and whichare not and thus must rely on other estimation techniques. However,illumination sources often vary with respect to how much infraredradiation they emit. Having an estimate of how much infrared radiationis present in the scene compared to visible illumination can provideinformation about the nature of the illuminant. For example,incandescent lamps emit visible light accompanied with strong infraredcontent. Office fluorescent lighting typically emits very littleinfrared when compared to scene luminance. Outdoor daylight containsmoderate amounts of infrared compared to scene luminance. For thereasons stated above, and for other reasons stated below which willbecome apparent to those skilled in the art upon reading andunderstanding the present specification, there is a need in the art fordetecting ambient infrared in a sensor equipped with an IR cut-offfilter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a cross-sectional view of one embodiment of a portion of animage sensor.

FIG. 2 shows a top view of one embodiment of an image sensor.

FIG. 3 shows a top view of the image sensor that is beneath the infraredfilter.

FIG. 4 shows a flowchart of one embodiment of an IR sensing method.

FIG. 5 shows a flowchart of one embodiment for determining IR_(avg).

FIG. 6 shows a block diagram of one embodiment of an imaging system ofthe present invention.

FIG. 7 shows a graph of pixel response for the IR-sensitive andIR-shielded pixels of an image sensor.

DETAILED DESCRIPTION

In the following detailed description of the invention, reference ismade to the accompanying drawings that form a part hereof and in whichis shown, by way of illustration, specific embodiments in which theinvention may be practiced. In the drawings, like numerals describesubstantially similar components throughout the several views. Theseembodiments are described in sufficient detail to enable those skilledin the art to practice the invention. Other embodiments may be utilizedand structural, logical, and electrical changes may be made withoutdeparting from the scope of the present invention. The followingdetailed description is, therefore, not to be taken in a limiting sense,and the scope of the present invention is defined only by the appendedclaims and equivalents thereof.

FIG. 1 illustrates a cross-sectional view of one embodiment of an imagesensor. For purposes of clarity, not all of the layers are shown in thisfigure. For example, there may be metal interconnect layers formedbetween the layers shown as well as dielectric layers for insulationpurposes.

The sensor is comprised of a substrate 130 that incorporates a pluralityof pixels or photodiodes 101-104. The photodiodes 101-104 areresponsible for converting light into an electrical signal for use bythe circuitry that reads the photodiode information. The higher theintensity of the light that strikes the photodiode 101-104, the greaterthe charge collected.

Optionally, a color filter array (CFA) 112 is positioned over thephotodiodes 101-104. This layer comprises the different color filtersrequired for the color system that is used. For example, the filters maybe red 107, green 106, and blue 108 for an additive RGB system or cyan,yellow, magenta, and an extra green for a subtractive CYMG system. Eachfilter color separates out that particular color for that photodiode. Inone embodiment, the color filters are formed in a “Bayer” color filterpattern that is well known in the art.

An IR cutoff filter 120 is positioned over the CFA 112. This filterblocks IR light from reaching certain ones of the photodiodes 101-104such that only visible light reaches the desired photodiodes 101-104. Atop view of the embodiments of the IR cutoff filter 120 of the presentinvention are illustrated in FIGS. 2 and 3 that are describedsubsequently.

A lens 113 is positioned over the IR cutoff filter 120. The lens 113 isresponsible for focusing light on the photodiodes 101-104.

Optionally, a plurality of micro-lenses 150 can be formed over thephotodiodes 101-104. Each micro-lens 150 can be formed over acorresponding photodiode 101-103. Each micro-lens 150 focuses theincoming light rays onto its respective photodiode 101-104 in order toincrease the light gathering efficiency of the photodiode 101-104.

FIG. 2 illustrates a top view of one embodiment of the IR cutoff filter203 over an image sensor. In this embodiment, the IR filter 203 isdeposited directly on the sensor 200 or its protective cover in apatterned fashion. The active array pixel filter 203 blocks IR fromreaching the active pixel array under the filter 203. A slot 205 isformed in the periphery of the filter 203 over the array to allow IR toreach the pixels under the slot 205. Optically “black” IR-sensitive(OBIR) pixels 300 as shown in FIG. 3 are formed around the periphery ofthe sensor 200. The OBIR pixels 300 are covered with a filter thatblocks all visible wavelengths of light from hitting the pixels but istransparent to IR illumination.

The slot 205 of FIG. 2 is only one embodiment of the IR cutoff filter.The opening or openings in the IR filter can be of any shape that can beformed into the filter, thus exposing a predetermined quantity of pixelsto IR illumination. For example, a slot can be positioned along only oneside of the sensor.

FIG. 3 illustrates the composition of the sensor pixel array lyingdirectly beneath the IR filter. This figure shows the regular activearray of pixels 301 that is covered by an IR cutoff filter. This array301 is surrounded by a ring of OBIR pixels 300 that must be under theopening in the IR filter. The OBIR pixels 300 are separated from theactive pixel array 301 by a ring of barrier pixels 305. The barrierpixels 305 shield the active pixel array 301 from IR leaking intoperiphery active pixels in the active array 301 due to the longwavelength of IR. The barrier pixels 305 may be shielded from bothvisible and IR illumination (using a metal layer) and also act tosubstantially stop any charge leakage from the OBIR pixels 300 into theactive pixels 301. This avoids contaminating the response of activepixels by IR generated charge.

The OBIR pixels 300 are the same type of pixels that are illustrated inFIG. 1 but are not covered by the IR cutoff filter as is the activearray of pixels 301. This allows the OBIR pixels 300 to receive IRillumination.

FIG. 3 also includes optically and IR black (OIRB) pixels 303 that areshielded from all light, both visible and IR. This can be accomplishedby a metal layer. These pixels 303 may or may not be under the slot inthe IR filter. The active pixel array 301 is covered with the patternedarray of color filters (e.g., Bayer CFA).

FIG. 4 illustrates a flowchart of one embodiment of a method for IRdetection in accordance with the image sensors of FIGS. 2 and 3. Thismethod reads the light signal from the OBIR pixels 401. These signalsare averaged and normalized by analog gain 403 to obtain IR_(avg) asdescribed later with reference to FIG. 5.

After the averaging and normalization of the OBIR signals 403, the lightsignals from the active pixels adjacent to the IR slot shown in FIG. 3are analyzed 405. These signals are averaged and normalized 407 byanalog gains to generate G_(avg). G_(avg) is then used to calculate theratio 409 of G_(avg)/IR_(avg).

Averaging the signals is accomplished by the equation:G _(avg)=(avg(P _(ij)green1)/GainG1+avg(P _(ij)green2)/GainG2)/2/T_(int activepixels)  (1)where ‘P_(ij) green1’ is a light signal from a pixel located in row iand column j in the sensor array, covered by a green CFA filter andlocated in such rows i that contain pixels covered with green and redCFA filters, ‘P_(ij) green2’ is a light signal from a pixel located inrow i and column j in the sensor array, covered by a green CFA filterand located in such rows i that contain pixels covered with green andblue CFA filters, ‘GainG1’ is the gain applied to the green1 pixels,‘GainG2’ is the gain applied to the green2 pixels, and‘T_(int activepixels)’ is the integration time the active pixels arereceiving.

During a normalization process, GainR, GainB, GainG1, and GainG2 for theBayer pattern are the four typical gains of an image sensor. The gainsare applied to corresponding pixels in the Bayer pattern. Since OBIRpixels may only differ from regular pixels by having a black filter, thered, green1, green2, and blue pixel gains keep getting applied to OBIRpixels as well in order to simplify sensor design.

Referring again to FIG. 4, after the averaging and normalizing of theOBIR signals 403, the light signals from the active pixels adjacent tothe IR slot shown in FIG. 3 are analyzed 405. These signals are thenaveraged and normalized 407 by analog gain to generate G_(avg).IR_(avg.) is discussed subsequently with reference to FIG. 5. IR_(avg.)is then used to calculate the ratio 409 of G_(avg.)/IR_(avg.). Alternateembodiments can calculate other ratios such as Y_(avg)/IR_(avg) whereY_(avg) is the average luminance of pixels after demosaicing and,optionally, color correction.

FIG. 5 illustrates a flowchart of one embodiment for determiningIR_(avg.). The method begins with zeroing a row counter, r 501. Thesignals from the OBIR pixels are then read 503. These signals are usedto determine four averages of P_(ij) from the OBIR pixels and areaffected by Blue/Red/Green1/Green2 gain. These averages are denoted asIR_(avgR), IR_(avgB), IR_(avgG).

IR_(avgG), IR_(avgB) and IR_(avgR) are calculated as follows.IR _(avgG)(r)=avg(avg(P _(ij)green1)/GainG1, avg(P_(ij)green2)/GainG2)/T _(int IR)  (2)IR _(avgR)(r)=avg(P _(ij)red)/GainR/T _(int IR)  (3)IR _(avgB)(r)=avg(P _(ij)blue)/GainB/T _(int IR)  (4)where P_(ij) are OBIR pixels from the r^(th) inner pixel ring asdetailed below. T_(int IR) is the integration time OBIR and OIRB pixelsare receiving, which is typically equal to T_(int) active pixels. Ifpixel ring r contains no pixels of type green1 or green2, thecorresponding term is excluded from the calculation of the average in(2).

These averages are compared to a saturation threshold 507, T_(sat). Thisthreshold is determined by the maximum pixel value D_(max) that can beread multiplied by a saturation constant, K_(sat)(i.e.,T_(sat)=K_(sat)*D_(max)). For example, a maximum pixel value of 1023multiplied by a constant of 18%. If ring r contains no pixels affectedby the blue or red gains, the comparison for the average value of thattype of pixel is not performed.

If none of the averages are less than T_(sat) 507, the pixels beingconsidered are all close to IR saturation. Therefore, “more inner” OIRBpixels that are further from the OBIR pixels are then read 509. The“more inner” OIRB pixels are read by incrementing the row counter 511and repeating the OBIR pixel read 503. This is repeated untilnon-saturated pixels are found. Since these pixels are shielded fromvisible and IR illumination, the signal generated by these pixels is duelargely to charge and light leakage from adjacent OBIR pixels, see FIG.7. Hence such signal may be substantially attenuated compared to OBIRpixel response thus allowing to assess IR illumination even when OBIRpixels are overexposed.

Charge leakage is the result of the fact that infrared photons areabsorbed at a higher depth than the visual wavelength photons due tosolid state sensor construction typically having a longer average traveldistance before the infrared photo-generated carriers are collected.Thus, the infrared-generated carriers can leak under the optical shieldof the OIRB pixels. The OIRB pixels collect the infrared-generatedcharge and their response is increased. Illumination free of infraredwill have no effect on OIRB pixel response. The magnitude of thatcomponent can then be determined in comparison to the visible lightillumination of the edge pixels.

OIRB pixels are particularly effective in cases where levels of IRradiation are so high as to saturate responses of OBIR pixels. Forexample, as the image sensor operates, the exposure for active pixels istypically selected to result in capture of visually pleasing images.Since OBIR pixels differ from active pixels only by having a differentCFA filter, their exposure would typically be identical to that of theactive pixels. This is true unless, optionally, the sensor isspecifically designed to allow setting exposure of OBIR pixelsindependently from active pixels. In such an embodiment, the sensorwould require a more complicated design.

When active pixels are exposed in a way optimal for image capture, OBIRpixel response can be saturated if the scene illuminant has a high IRcontent. In such case, values of OIRB pixels should be read out insteadof OBIR.

After non-saturated pixels are found, the average of the averages thatare less than the saturation threshold T_(sat) is determined 513. Thisis determined by the following equation where, typically,T_(intIR)=T_(intactivepixels) and K_(ir) is an IR sensitivitycoefficient:IR _(avg)(r)=avg(IR _(avgR)(r), IR _(avgB)(r), IR _(avgG)(r))/K_(ir)(r)  (5)In the formula above, if IR_(avgR), IR_(avgG), and/or IR_(avgB) are lessthan T_(sat), or pixels of a certain type (e.g., affected by the blue orred gains) do not exist in ring r, these averages are excluded from theaverage.

If IR coefficient is set to 1 for OBIR, K_(ir)(0) is defined as 1 forOBIR pixels, K_(ir)(0)=1. K_(ir) for inner OIRB pixels is typicallysubstantially less than 1. For example, the inner row of OIRB pixelscould have K_(ir)(1)=0.05. The next row, K_(ir)(2) might haveK_(ir)(2)=0.0025. The coefficient would get progressively smaller due toreduced sensitivity of inner OIRB pixels. K_(ir)(r) can be determinedusing a calibration procedure.

One such calibration procedure illuminates the sensor with IR that hasspectra expected to be encountered in actual operation. A flat field isthen imaged. The sensor exposure and gains are adjusted to makeresponses of OIRB pixels equal to a constant (e.g., T_(sat)). ExposureT_(int)(r) is recorded for r=0, as T_(int)(0). Gains are then frozen.For each inner ring of OIRB pixels, exposure is adjusted to make theirresponses equal to T_(sat). The OIRB pixels probably saturate in thiscondition. T_(int)(r) is recorded for r=1, 2, etc. If K_(ir)(0)=1,K_(ir)(r)=T_(int)(0)/T_(int)(r).

The embodiment of the sensor of FIGS. 2 and 3 is optimized for useduring manufacture of a sensor array. The opening in the IR cutofffilter can be added when the IR filter is formed over the sensor array.

The IR detection embodiments can also be used when a sensor arrayalready has an IR cutoff filter covering the entire array of pixels.Such an IR detection embodiment relies on the fact that spectraltransmittance of reflective IR-cutoff filters is highly angle dependent.Such filters allow a small amount of IR illumination through whenilluminated at an angle other than perpendicular to the filter plane.

One problem solved by the present embodiments in IR sensing in using avisible light sensor is that the level of IR can substantially differfrom the level of visible illumination, sometimes by many orders ofmagnitude. By using OBIR and OIRB pixels the IR is sensed in ahigh-dynamic-range fashion. The present embodiments require only a fewchanges to the sensor such as forming a black filter over some pixels, ametal layer over other pixels, and forming “barrier” pixels.

There are also alternative embodiments for IR sensing. For example, theexposure of IR pixels can be controlled independently from activepixels. In this embodiment, the exposure is varied to avoid overexposureof IR pixels and the OIRB pixels are optional.

In yet another embodiment, the method for IR sensing is similar to theembodiment of FIGS. 2-5 but not all of the OIRB pixels are shieldedcompletely. The pixels adjacent to the OIRB pixels have a certain area(e.g., 50%) covered by metal. The next inner row is covered by metal toa greater extent (e.g., 75%) and the next row is covered to an evengreater amount (e.g., 100%).

In still another embodiment, the OIRB pixels are not used. In thisembodiment, special high dynamic range pixels are used to sense IR.

FIG. 6 illustrates a block diagram of one embodiment of an imagingsystem of the present invention. The system comprises an image sensor600 as described previously, coupled to a control circuit 601. Thissystem can represent a camera or some other imaging device.

In one embodiment, the control circuit 601 is a processor,microprocessor, or other controller circuitry that reads and processesthe image from the image sensor device 600. For example, the imagingsystem can be a digital camera in which the image sensor device 600 isexposed to an image for recording. The control circuitry 601 executesthe above-described embodiments and reads the accumulated charges fromthe photodiodes of the image sensor device 600. The control circuitry601 may then process this data into an image or simply store the data.

The data from the system can be output to other systems over an I/Ocircuit 603. The I/O circuit 603 may be a Universal Serial Bus (USB) orsome other type of bus that can connect the imaging system to a computeror other system.

CONCLUSION

In summary, the embodiments discussed herein provide a way to measurethe amount of infrared illumination that reaches an image sensor.Knowing the amount of infrared illumination is useful in automatic whitebalance algorithms. A large amount of infrared illumination informs thewhite balance algorithm that the scene is illuminated by an incandescentlight source. The white balance algorithm can then shift the colortemperature of the received image to adjust for the extra red colorcast. A low amount of infrared informs the white balance algorithm thatthe camera is imaging a scene illuminated by a fluorescent light source,such as office lighting, and the color temperature of the received imagemust be shifted to remove a greenish cast. A moderate amount of infraredindicates that the camera is used outdoors in daylight conditions. Inthis condition, the white balance would typically remove a blue colorcast from the image, if any.

Although specific embodiments have been illustrated and describedherein, it will be appreciated by those of ordinary skill in the artthat any arrangement that is calculated to achieve the same purpose maybe substituted for the specific embodiments shown. Many adaptations ofthe invention will be apparent to those of ordinary skill in the art.Accordingly, this application is intended to cover any adaptations orvariations of the invention. It is manifestly intended that thisinvention be limited only by the following claims and equivalentsthereof.

1. An image sensor device comprising: an array of pixels; an infraredcutoff filter substantially covering a first predetermined area of thearray of pixels; and means for inhibiting both infrared and visiblelight spectrum substantially covering a second predetermined area of thearray of pixels wherein a third predetermined area of the array ofpixels remains exposed to both visible and infrared illumination.
 2. Thedevice of claim 1 wherein the infrared cutoff filter reducesillumination of the first predetermined area of the array of pixels inthe infrared wavelength.
 3. The device of claim 1 and further comprisinga color filter array formed over the array of pixels.
 4. The device ofclaim 3 wherein the color filter array is formed in a Bayer pattern. 5.The device of claim 1 wherein the means for inhibiting is a layer ofmetal.
 6. The device of claim 1 wherein the means for inhibiting is ablack color filter array.
 7. The device of claim 1 wherein the thirdpredetermined area substantially encircles the first predetermined area.8. The device of claim 7 wherein the second predetermined areasubstantially encircles the third predetermined area.
 9. An image sensordevice comprising: a substrate comprising an array of photodiodes; acolor filter array formed over the array of photodiodes such that adifferent color filter is formed over each photodiode; an infraredcutoff filter formed over a predetermined area of the array ofphotodiodes; an infrared barrier formed substantially around a peripheryof the first predetermined area; a plurality of infrared photodiodesformed substantially around a periphery of the infrared barrier; and aplurality of black photodiodes formed substantially around a peripheryof the plurality of infrared photodiodes.
 10. The device of claim 9wherein the substrate is comprised of p+ silicon and each photodiode iscomprised of n− silicon.
 11. The device of claim 9 wherein the substrateis comprised of n− silicon and each photodiode is comprised of p+silicon.
 12. The device of claim 9 wherein each of the plurality ofinfrared diodes comprises a color filter formed to selectively admitpredetermined wavelengths of light to the photodiode and all of theplurality of infrared photodiodes is not blocked from exposure to andinfrared wavelength of light.
 13. The device of claim 12 wherein each ofthe plurality of black photodiodes comprises a filter that substantiallyinhibits admittance of all light wavelengths except infrared.
 14. Amethod for fabricating a solid state image sensor device, the methodcomprising: forming an array of photodiodes in a substrate; forming acolor filter array over the array of photodiodes; forming an infraredcutoff filter over a first predetermine area of the array ofphotodiodes; and forming a black filter over a second predetermined areaof the array of photodiodes such that a third predetermined area of thearray of photodiodes is filtered only by the color filter array.
 15. Themethod of claim 14 and further including forming an infrared barrierbetween the third predetermined area and the first predetermined area.16. The method of claim 14 wherein the third predetermined area isformed substantially around a periphery of the first predetermined areaand the second predetermined area is formed substantially around aperiphery of the third predetermined area.
 17. The method of claim 14wherein forming the black filter comprises forming a layer of metal overthe second predetermined area.
 18. The method of claim 14 and furtherincluding forming a lens over the infrared filter.
 19. An image systemcomprising: control circuitry that generates an image in response topixel information; and an image sensor device, coupled to the controlcircuitry, for generating the pixel information in response to receivedlight, the device comprising: an array of pixels; an infrared cutofffilter substantially covering a first predetermined area of the array ofpixels; and means for inhibiting visible light spectrum substantiallycovering a second predetermined area of the array of pixels wherein athird predetermined area of the array of pixels remains exposed to bothvisible and infrared illumination.
 20. The system of claim 19 whereinthe image system is a digital camera.